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1 This application is submitted in the name of the following inventor: 

2 

3 Inventor Citizenship Residence Address 

4 Cohen, Earl United States 43750 Cameron Hills Drive 

5 Fremont, CA 94539 
6 

7 The assignee is Cisco Technology, Inc. , a California corporation having an 

8 office at 1 70 West Tasman Drive, San Jose, California 95 1 24. 

-9 

□0 Title of the Invention 

42 Route/Service Processor Scalability Via Flow-Based Distribution of Traffic 

Jj4 Background of the Invention 

Is 

16 J. Field of the Invention 
17 

1 8 The present invention relates to architecture for a router. More particularly, 

1 9 the present invention relates to architecture for a router with scalable processing power. 

20 
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1 2. The Prior Art 
2 

3 Computers generate and utilize large amounts of data. Different compo- 

4 nents in a computer network, for example, several different computers each with an 

5 internet connection or "uplink", need to be linked together to allow for the transfer of 

6 data among the different components in the network. Routers perform this function. 

7 Routers take data input from one component in a network and ensure that it is properly 

8 transferred to another component in the network. 

ttO Different components in a computer network may also generate data in dif- 

i|l ferent forms and at different speeds. A router system in conjunction with other network 

12 interface components can properly bundle different types of information, transmit the in- 

S]3 formation among different components in a computer network, and ensure that each 

;T4 component of the computer network is allowed to give and receive data at a rate proper 

5§5 for that particular component. Examples of network interface components include 

16 Ethernet transceivers and CT3 HDLC channel link data managers (as embodied in the 

1 7 PMC-Sierra CT3 interface chip). 
18 

19 Data bytes are generally bundled into "packets", a discrete grouping of in- 

20 formation bytes that will be transferred together along the router system. Packets con- 

21 taining related information in a logical order are grouped into "flows". A flow consists of 

22 a unidirectional stream of packets to be transmitted between a particular source and a 
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1 particular destination. Packets within the same flow will have the same 

2 source/destination address, the same source/destination port, and the same protocol. It is 

3 required to maintain the packet ordering within a flow. 
4 

5 Speed and accuracy are two important features of routers. If the data trans- 

6 fer rate, or throughput rate, is not rapid enough, the end user of the required data must 

7 wait for it to arrive, wasting valuable processing time. Also, input and output queues 

8 waiting to transfer data may overflow if information is being added more rapidly than it 
q9 is being removed. Accuracy in transfer is also important to ensure that data arrives at its 
C$0 proper destination in the proper order. 

^2 Routers can also provide services as they perform the data transfer or 

■43 switching. Some of these services involve gathering information about the data being 

r|4 transferred or performing some other processing function upon the data as it is being 

■A"- 

§5 transferred from one component in the computer network to another. Some of the more 

16 recent service developments involve "touching", or performing some processing func- 

17 tion, on the data in most or every packet as it is transferred. Such "high touch" services 

18 can require the router system to have a great deal of processing power. New services are 

19 being developed all the time, and thus the amount of processing that performing services 

20 requires is unbounded. 
21 
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1 Current routing systems generally operate on a "one processor per line 

2 card" model. A line card is a group of components in a computer network such as a 

3 group of modems or an internet connection which sends and receives data. In a "one 

4 processor per line card" model, all of the processing for a single line, or a group of 

5 related lines, is done by a dedicated processor. This model works acceptably well if the 

6 line's rate of data input, or input interface rate, is low enough so that the processor does 

7 not become a bottleneck in the system. With low enough interface rates, one processor 

8 can handle multiple lines. 
q9 

WO However, the line's interface rate may increase as higher-speed computers 

^1 or internet connections are added. Furthermore, additional router services may become 

■4:2 desirable. Eventually, the dedicated processor may no longer be able to keep up with the 

43 processing requirements of the line and the throughput rate will slow down. If the 

Q4 throughput rate becomes unacceptably slow, the only solution is generally to replace the 

[|j5 dedicated processor with a more powerful machine. This is an expensive solution. In 

16 certain cases, for example a high-speed internet line or "uplink", a processor may not 

17 even be available that can keep up with the interface rate of the high-speed line. Current 

1 8 internet uplink rates such as Gigabit Ethernet (1 gigabit per second) are already becoming 

1 9 difficult to handle with a single dedicated processor. 
20 

21 Accordingly, it would be desirable to provide a router architecture that 

22 would be easily scalable to accommodate future router service growth as well as expand- 
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1 able uplink connections, and which would allow processors to be added as more proc- 

2 essing power became necessary, without requiring the entire system's future hardware 

3 needs to be purchased initially. It would also be advantageous to maintain packet order- 

4 ing within each flow while still flexibly routing among different processors as necessary. 

5 

6 Summary of the Invention 

7 

8 The invention provides a system and a method for a router architecture that 

r 9 is scalable, that is, as more processing power is desired, more individual processors can 
§0 be added. The data flow from each line can be distributed among all of the processors in 
11 the system. As desired services are added, increasing the amount of "touch" or process- 
th. ing performed on the packets in the system, more processors can be added to carry the 
33 increased load. The router architecture is also able to distribute the high interface rate of 
H A an uplink connection in the same manner. 

16 In a preferred embodiment, packets are allocated to processors in a manner 

1 7 that allows the original order of data packets within the same flow to be maintained. The 

18 system uses a hash function to distribute the flows, making sure that packets within the 

19 same flow are sent to the same processor so that the original packet order in each flow is 

20 maintained. Different flows may be sent to different processors. 
21 
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Figure 1 shows a logical block diagram of the router architecture of the pre- 
sent invention. 

Figure 2 shows a sample flow diagram for a data packet. 

Figure 3a shows an example port adapter design for an Ethernet interface. 

Figure 3b shows an example port adapter design for a channelized T3 line 
(CT3) interface. 

Figure 4 shows an example Data Compiler. 

Figure 5 shows an example packet digest format. 

Detailed Description of a Preferred Embodiment 

In the following description, a preferred embodiment of the invention is 
described with regard to preferred process steps and data structures. However, those 
skilled in the art would recognize, after perusal of this application, that embodiments of 
the invention may be implemented using a set of general purpose computers operating 
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1 under program control, and that modification of a set of general purpose computers to 

2 implement the process steps and data structures described herein would not require undue 

3 invention. 
4 

5 The present invention provides a router architecture capable of implement- 

6 ing a wide variety of services while balancing the router system load among a number of 

7 processors. The router architecture of the present invention is also capable of distributing 

8 the load from a single uplink among the multiple processors. The ability to distribute the 
?*£ router system load among all of the processors in the system makes the system through- 
30 put scalable; the throughput rate increases with each additional processor. Because there 
Ml is no cooperation problem among the processors and each is acting independently and 
32 preferably on separate flows, the system throughput is expected to scale approximately 
3P3 linearly with the number of processors. 

H4 

^5 The use of multiple processors which are redundant also provides added 

16 reliability. If one processor fails, the processing tasks can be allocated among the 

1 7 remaining processors . 
18 

19 FIG. 1 shows a logical block diagram of the components in the router sys- 

20 tern. FIG. 1 is intended to show the interconnections and resulting scalability of the 

21 router system architecture, and is not meant to be an accurate representation of the num- 

22 ber of the various components in the router system in any particular embodiment. Indi- 
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1 vidual data interface components 10 are connected to a set of Port Adapters ("PA"s) 20, 

2 which are in turn connected to a set of Data Compilers (DCs) 30. The DCs 30, an 

3 Uplink 60, and a set of Route Processing Engines (RPEs) 50 are all interconnected 

4 through an Xbar 40. The Uplink 60 is connected to an external network such as the 

5 internet 70. Data transfer originates at either the individual data interface components 10 

6 or the internet 70 and travels through the router system to the Xbar 40, where it is sent on 

7 to be processed by a selected one of the RPEs 50. Once processing is complete, the data 

8 is transferred back through the Xbar 40 to its final destination, either the internet 70 or an 
□9 individual data interface component 10. 

go 

Si Data flow through the system is more easily followed through a sample 

; kl flow diagram for a data packet as shown in FIG. 2. FIG. 2 shows only one example of a 

;J3 data packet flow path; numerous other paths are possible using a router system of the pre- 

■ 14 sent invention. FIG. 2 shows an individual computer generating the original data. This 

i5 data may be sent out via modem to be routed to another part of the router system. Meth- 

16 ods of transferring data via modem are well known in the art of computer networking 

17 systems. The example in FIG. 2 of an individual computer and a modem is purely illus- 

18 trative of an individual data interface component 10 in a router system as shown in 

19 FIG. 1. Other types of data interface components could serve the same function; for ex- 

20 ample, a Tl connection (1.5 Mbps) could also function to provide data to a PA 20. 
21 



CIS-032-B Application 010698 



CIS-032-B 

1 In step 110, the data is first sent to the PA 20. A single PA 20 is capable of 

2 handling the throughput from multiple modems or other individual data interface compo- 

3 nents 10 since such components send data at a relatively slow and intermittent rate. The 

4 PA 20 bundles this data it has received from multiple modems or other interface lines 

5 into packets. Each packet is represented as a series of descriptors in a format compatible 

6 with the PA 20. Network interface components, such as many Ethernet transceivers and 

7 HDLC channel managers, use a descriptor approach to transfer packet data to a local 

8 memory, possibly with the help of a local microprocessor. Each descriptor effectively 
r 9 indicates a memory block (called a particle) and a size, as well as control information 
30 such as whether this is the first or last descriptor in a packet. These packets are the main 
^ 1 units of data sent across computer network systems. The creation and use of data packets 
-|2 are well known in the art of computer networking systems. 

I 3 

irf 4 Various types of PAs are available to convert between different user inter- 

a|5 faces and a generic PCI bus. The PCI (Peripheral Component Interconnect) bus is known 

16 in both the computer and networking industries. FIG. 3a shows a representative PA de- 

17 sign for an Ethernet interface. FIG. 3b shows a representative PA design for a channel- 

18 ized T3 line (CT3) interface. Other user interfaces such as OC-3 (a fiber-optic standard) 

19 may also be converted using a PA. 
20 

21 In step 1 12, the data packets from the PA 20 are sent to a DC 30. A DC 

22 provides an interface between the internal fabric in the router and a generic PCI bus 
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1 which can connect to a PA. Figure 4 shows a representative Data Compiler. The DC 30 

2 further prepares the packets for transfer along the routing system by putting them into 

3 packet digest form, and then forwards the packets along to the Xbar 40. A packet digest 

4 contains both a header and a payload. The DC 30 will further "packetize" the incoming 

5 packets by converting them from the particles to a consecutive block of memory, forming 

6 the "payload" for a packet digest. The DC 30 will also add a packet digest header. 
7 

8 The packet digest header contains information about the packet to aid in the 

p9 prompt forwarding and processing of the packet, such as — the Xbar 40 source port num- 

."fjo ber, the size of the packet digest header and payload, the destination RPE 50, the packet 

fljl priority, the protocol type, and the class of service required. 

-h 

Mp Figure 5 shows an example packet digest format. In a preferred embodiment, the 

;14 packet digest includes the following fields: 

I 5 

16 o a CMD field to control features of the packet digest (such as the TIMESTAMP field, as 

17 described below), and to indicate a type of packet digest (where there is more than one 

1 8 type, such as in alternative embodiments); 
19 

20 o a SRC PORT field to indicate a source ID of the transmitting entity; 
21 

22 o a PKT_OFF field and a NE W_PKT_OFF field to indicate an number of bytes of padding 

23 between the packet digest header and the packet digest payload (where the PKT_OFF in- 
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1 dicates a total size of the packet digest header plus any padding, while the 

2 NEW PKT OFF field indicates a new value for PKT_OFF); 

4 o a PDH_SIZE field to indicate the size of the packet digest header; 
5 

6 o a DEST_OFF field to indicate an offset of the destination IP address in the packet digest 

7 header; 
8 

9 o a "FIB Leaf field to indicate a pointer to the leaf in a FIB lookup m-trie for the destina- 

4j0 tion IP address; 

:*J2 o a DEST_MASK field to indicate which Xbar ports are intended as destinations for the 

"43 packet (where a selected bit <39> of the DEST MASK field preferably indicates a multi- 

□4 cast packet); 
M5 

j 6 o a QUEUE field to indicate to which hardware output queue at the particular output 

17 interface to direct the packet; 
18 

19 o an INTERFACE field to indicate to which hardware output interface to direct the 

20 packet; 
21 

22 o a TIMESTAMP field to indicate a timestamp or other data to be written by the in- 

23 bound Xbar interface (where a selected bit CMD <0> indicates that the TIME- 

24 STAMP field should be written to); and 



CIS-032-B Application 010698 1 1 



CIS-032-B 

1 

2 o a set of additional words reserved for software classification data. 
3 

4 In order to determine the destination RPE 50, the DC 30 includes a classifi- 

5 cation engine coupled to the crossbar interface to determine which RPE 50 will receive 

6 each packet. Packets are not distributed randomly among RPEs, so as to maintain the 

7 original ordering of packets within the same flow. The classification engine distributes 

8 packets so that packets within the same flow will be sent to the same RPE, but packets in 
D9 different flows may be sent to different RPEs and possibly reordered as determined by 
3o RPE resources. 

I 1 

-|2 The classification engine analyzes the router system traffic and distributes 

5b flows to multiple processors. The distribution is partly table-driven, and is thus flexible 

Q4 and easily changeable, even dynamically. In a preferred embodiment, the classification 

935 uses a hash function of packet flow information to select a specific RPE 50 to handle 

16 processing for that packet flow. The hash function distributes packets evenly among the 

17 processors in response to flow information such as the source/destination address, the 

18 source/destination port, and the protocol. The hash function can operate using any in- 

1 9 formation that will allow for flow preservation. 
20 
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1 In a preferred embodiment, the hash function performs an XOR (exclusive 

2 OR) logical function of several bytes of fields from the IP header of the packet, including 

3 the following fields: 
4 

5 o a source IP address (four bytes); 
6 

7 o a destination IP address (four bytes); and 
8 

o9 o a protocol type value (one byte). 

qo 

S 1 The XOR logical function is performed for all nine bytes, providing a sin- 

i|2 gle byte output value, which can be used to index into a 256-entry table for selecting the 

1 3 RPE50. 

-SjjS Once the DC 30 has determined the packet's destination RPE 50 and the 

16 packet header and pay load are ready to be sent out, the packet (now in packet digest 

17 form) is placed in an input queue to be stored until it is sent to the selected RPE 50. Note 

18 that the amount of processing performed by the DC 30 on each packet is bounded; the 

19 DC 30 only does a fixed amount of work per packet. There is no cost in terms of DC 30 

20 processing for adding additional services, as all such additional processing will be per- 

21 formed by the RPEs 50. Thus, the DC 30 is prevented from becoming a system bottle- 

22 neck as more services are added. In the router system of the present invention, DC 30 
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1 input queues should only overflow if there are not enough RPEs 50 to support the re- 

2 quested level of services at the incoming data rates; if input queues overflow, it is always 

3 possible to add more RPEs 50. 
4 

5 In step 1 14, packets are sent from the DC 30 to the Xbar 40. The Xbar 40 

6 provides the interconnection between the data origination component ports, the router 

7 system processor ports, and the data destination component ports. In one data transfer, 

8 the data packet will cross through the Xbar 40 twice (step 1 14 and step 1 18 in FIG. 2). 
^9 The Xbar 40 receives an incoming packet and transfers it to the selected RPE, and then 
B0 the Xbar 40 transfers the packet back to its destination port. The Xbar 40 must provide 
^1 enough bandwidth to handle the data interface rates of both the DCs 30 and the 
-J 2 Uplink 60. Xbar interconnections are known in the art of computer and router systems. 
03 

:i4 The Xbar 40 includes a number of ports; each port is a pair of uni- 

j| 5 directional links. The Xbar 40 implements a simple arbitration scheme between sending 

16 and receiving components, ignoring whether the destination port has the resources to 

17 handle the packet or not. Packets can be dropped at any point if there are not sufficient 

18 resources to handle them. Software must ensure that there are sufficient resources avail- 

19 able at a destination port, for example, memory space at the output to queue the packet, 

20 before sending a packet over the Xbar 40. Output queuing protocols determine what oc- 

21 curs when an output queue overflows; output queue management is performed at the 

22 DC 30. 
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The Xbar 40 includes an arbiter — ports forward their requests to the arbi- 
ter, and grants are sent back. The Xbar 40 arbiter uses a windowing scheme in which 
each port can make requests for multiple destinations at the same time; the arbiter at- 
tempts to find the best fit between requesting ports and destinations. The Xbar 40 can 
also support multicast — a port can request that a packet be sent to multiple destinations. 
In the preferred embodiment, the Xbar 40 waits for all ports to be free, and then sends the 
multicast request all at once. Numerous other Xbar multicast schemes are well known in 
the art. 

The Xbar 40 in the preferred embodiment discussed herein is packet-based, 
but the Xbar 40 could be designed to be cell-based and still fall within the inventive con- 
cepts disclosed herein. It is also possible to provide multiple Xbars or a bus that per- 
forms the same function as the Xbar. The methods discussed herein for the implementa- 
tion of such an Xbar device are by no means limiting. Alternate methods for the imple- 
mentation of such an Xbar device will readily suggest themselves to those of ordinary 
skill in the art. 

In step 116, the Xbar 40 transfers the packet to the destination RPE 50. 
There are multiple RPEs 50 connected to the Xbar 40 that provide both switching and 
services, including the high-touch type of services that may require a great deal of proc- 
essing power. Such services may also be added later to the system as more services are 
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developed and become available. Additional RPEs 50 can be added to the router system 
to provide additional processing power; this provides the scalability feature of the present 
invention. 

An RPE 50 receives a packet digest with a packet digest header containing 
classification information provided by the DC 30. Based on the classification informa- 
tion, the packet is put into one of a small number of input queues on the RPE 50. The 
intent of the RPE 50 queuing is to provide the capability for handling both high- and low- 
priority packets. Packets are processed from among the RPE's input queues by the 
RPE 50 processor in whatever order it sees fit. 

The RPE 50 processor will perform various tasks on the packet, including 
switching, tag application/update, access list processing (filtering), and all other router 
service functions. Some of these operations result in the RPE 50 modifying the packet 
digest, others only update RPE-internal data structures. When the RPE 50 processor is 
done with a packet, the RPE 50 will know the packet's final destination. The example 
destination in FIG. 2 is the internet 70 via the Uplink connection 60. However, it would 
be evident to one of ordinary skill in the art that the packet's destination could also be an 
individual data interface component 10 via a DC 30 and a PA 20. 

In step 118, the packet is put back on the Xbar 40 from the RPE 50. In 
step 120, the Xbar 40 transmits the packet to the Uplink 60. The Uplink 60 is a high- 
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1 bandwidth PA with its own DC. Like a DC, the Uplink 60 is connected to the Xbar 40 

2 via an Xbar link. However, in order to meet its bandwidth requirements, an Uplink 60 

3 may require multiple Xbar ports. The Xbar 40 can support multiple Uplink 60 

4 connections. 

5 

6 In step 122, the Uplink 60 transfers the data to the internet 70, completing 

7 the data transfer from the individual data interface component 10 to the internet 70. 
8 

r9 The Uplink(s) 60 can also send data to the Xbar 40 for transmission to an 

30 RPE 50 and eventual transfer to an individual data interface component 10. The 

£1 Uplink(s) 60' s traffic from the internet 70 is also distributed via the Xbar 40 to the vari- 

: 4j2 ous RPEs 50 using a hash function. In this way, the router system can support a plurality 

|3 of Uplink(s), each of arbitrary speed. 
jf4 

Cf5 Various alternate embodiments of the present invention have also been 

16 conceived, and would be clear to those skilled in the art after reviewing this application. 

17 For example, in alternative embodiments, the DC and a specific PA type can be inte- 

18 grated together to produce a more cost-effective, though less easily adaptable, interface. 
19 

20 Other alternative embodiments include performing some services of a 

21 bounded nature on the DCs, as is currently done with output queuing. For example, the 

22 DCs could accumulate statistics and do flow-based accounting on each packet. In this 
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1 way, the DCs could handle some portion of the known processing load leaving un- 

2 bounded and future services to the RPEs. 
3 

4 Yet other alternative embodiments include adding specialized processing 

5 engines connected to the Xbar. Packets can be sent through these specialized engines 

6 either before or after or instead of an RPE to perform services such as compression/ 

7 decompression, encryption, or routing. The classification engine on a DC could 

8 determine the type of service required by a packet and route it appropriately. 
|:;:9 

! : 10 While embodiments and applications of this invention have been shown 

3l and described, it would be apparent to those skilled in the art that many more modifica- 

$2 tions than those mentioned above are possible without departing from the inventive con- 

5:3 cepts herein. The invention, therefore, is not to be restricted except in the spirit of the 

:JL4 appended claims. 
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1 Claims 

2 

3 1 . A scalable routing system for distributing packets in a network, in- 

4 eluding 

5 a plurality of data compilers; 

6 a plurality of port adapters connected to said data compilers; 

7 a plurality of route processing engines; and 

8 at least one structure connecting said plurality of route processing engines 
r 9 to said data compilers. 

30 

2. The scalable routing system of claim 1, including at least one uplink 

32 connection to an external network connected to said at least one structure. 

□3 

Hf4 3. The scalable routing system of claim 1, wherein said at least one 

q| 5 structure includes a crossbar. 
16 

17 4. The scalable routing system of claim 1, wherein said at least one 

1 8 structure includes a bus. 
19 

20 5. The scalable routing system of claim 1, wherein said plurality of 

21 data compilers use a hashing function to distribute packet flows among said plurality of 

22 route processing engines. 
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1 

2 6. The scalable routing system of claim 2, wherein both said plurality 

3 of data compilers and said at least one uplink connection to an external network use a 

4 hashing function to distribute packet flows among said plurality of route processing en- 

5 gines. 

6 

7 7. The scalable routing system of claim 6, wherein said hashing func- 

8 tion is designed to maintain the original order of packets in the same flow while allowing 
r ;9 packets in different flows to be processed out of order. 

30 

=31 8. The scalable routing system of claim 6, wherein said hashing func- 

4_2 tion is designed to maintain the original order of packets in the same flow while allowing 

9 3 packets in different flows to be processed out of order. 

3 4 

f|5 9. The scalable routing system of claim 8, whereby processing power 

16 of said system can be scaled by adding additional route processing engines to said plural- 

17 ity of route processing engines. 
18 

19 10. The scalable routing system of claim 8, whereby the processing 

20 power of said system can be scaled by adding additional route processing engines to said 

21 plurality of route processing engines. 

22 
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1 11. A scalable routing system for distributing packets in a network, in- 

2 eluding 

3 a plurality of network interfaces; 

4 a plurality of route processing engines; 

5 a fabric interconnecting said plurality of network interfaces and said plural- 

6 ity of route processing engines; 

7 wherein each of said plurality of network interfaces uses a hashing function 

8 to distribute packet flows among said plurality of route processing engines; and 

r 9 wherein the processing power of the scalable routing system can be scaled 

QO by adding additional route processing engines to said plurality of route processing 

1 engines. 
^2 

03 12. The scalable routing system of claim 11, wherein said fabric 

H4 includes a crossbar. 

1 5 

16 13. The scalable routing system of claim 11, wherein said fabric 

17 includes a bus. 
18 

19 14. The scalable routing system of claim 11, wherein said hashing 

20 function is designed to maintain the original order of packets in the same flow, while 

21 allowing packets in different flows to be processed out of order. 
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1 15. The scalable routing system of claim 11, wherein said network 

2 interfaces include data compilers and port adapters. 
3 

4 16. The scalable routing system of claim 15, wherein said network 

5 interfaces include at least one uplink connection to an external network. 
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1 Abstract of the Disclosure 

2 

3 The invention provides a router architecture that is scalable, that is, as more 

4 processing power is desired, more individual processors can be added. The data flow 

5 from each line can be distributed among all of the processors in the system. As desired 

6 services are added, increasing the amount of "touch" or processing performed on the 

7 packets in the system, more processors can be added to carry the increased load. The 

8 router architecture is also able to distribute the high interface rate of an uplink connection 
t : 9 in the same manner. Packets are allocated to processors in a manner that allows the 
OO original order of data packets within the same flow to be maintained. The system uses a 
U 1 hash function to distribute the flows, making sure that packets within the same flow are 
s|2 sent to the same processor so that the original packet order in each flow is maintained. 
Mf 3 Different flows may be sent to different processors. 
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